Robot apparatus and method of controlling robot apparatus

ABSTRACT

A robot apparatus includes a grasping section that grasps an object, a recognition section that recognizes a graspable part and a handing-over area part of the object, and a grasp planning section that plans a path of the grasping section for handing over the object to a recipient by the handing-over area part. The robot apparatus further includes a grasp control section that controls grasp operation of the object by the grasping section in accordance with the planned path.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 14/961,128, filed on Dec. 7, 2015, which is acontinuation application of U.S. patent application Ser. No. 13/781,877,filed on Mar. 1, 2013, now U.S. Pat. No. 9,205,559, issued Dec. 8, 2015,and claims the benefit of priority from prior Japanese PatentApplication JP 2012-053108, filed Mar. 9, 2013, the entire content ofwhich is hereby incorporated by reference.

BACKGROUND

The technique disclosed in this specification relates to a robotapparatus, a method of controlling the same, and a computer program,which is engaged in activities in user's daily life in order to performlife support, such as household chores, welfare services, nursing care,assistance services, etc., for example. In particular, the techniquerelates to a robot apparatus, a method of controlling the same, and acomputer program, which performs handing-over interaction of an objectwith a user.

To date, robot apparatuses that are designed to carry out fixed tasksand installed in factories, etc., have been widespread. However, inrecent years, robot apparatuses capable of performing various kinds ofwork are being realized. A robot apparatus in the latter case is appliedto various fields, such as welfare services, nursing care, assistanceservices, for example, and brings a desired object for a physicallyhandicapped person, etc., in order to hand over the object to theperson.

However, even if a robot apparatus grasps an object in an easy way toreceive (or an easy posture of an object to receive), and tries to handover the object, a recipient user sometimes finds it difficult toreceive the object. Also, there are cases where an object to be handedover involves danger if handed over inadvertently, such as in the caseof a cutting tool, a cup of hot drink, etc.

For example, proposals have been made on a method and apparatus forhanding over goods, in which an easy way for a robot to receive goods ispresented to a person, and the goods are handed over from the person tothe robot by an easy way without using a visual device (for example,refer to Japanese Unexamined Patent Application Publication No.2007-21588). Also proposals have been made on a robot grasp controlapparatus which controls a robot arm so as to receive goods from aperson (for example, refer to Japanese Unexamined Patent ApplicationPublication No. 2005-219161). On the contrary, no proposal has been madeon the point of handing over goods from a robot to a person. Also, inthis method for handing over goods, it is necessary to attach a wirelesstag to the target goods, and further, it is necessary to get the goodscloser to or contacted a reading device to check a positionalrelationship, and to display the relationship to a person, and then tohand over the goods at an instructed position by the person.Accordingly, in order to achieve handing over of goods, it becomesnecessary to implement various devices and procedures, and thus it isthought that applicable cases are limited.

Also, a proposal has been made on a technique for modeling dailycommodities in consideration of comfortability of a recipient at thetime of grasping and handing over (for example, refer to Yamanobe, etal., “Grasping of daily commodities in consideration of comfortabilityof recipient at the time of handing over”, (the 29th Annual Conferenceof the Robotics Society of Japan (Sep. 7 to 9, 2011))). In thistechnique, descriptions have been given that information for eachprimitive, such as an operation section, a holding section, etc., of adaily commodity, such as a mug, etc., is described in a structuredescription language format, a priority of each primitive is determinedfor selecting a primitive, and a plan of a grasping position and postureshould be made after determining a position of an object inconsideration of influence given to the recipient. However, there hasnot been any specific disclosure on a method of making a grasping plan.

SUMMARY

It is desirable to provide an excellent robot apparatus, a method ofcontrolling the robot apparatus, and a computer program, which iscapable of performing handing-over interaction of an object with a userwithout imposing excessive load on the user and without endangering theuser.

According to an embodiment of the present disclosure, there is provideda robot apparatus including: a grasping section configured to grasp anobject; a recognition section configured to recognize a graspable partand a handing-over area part of the object; a grasp planning sectionconfigured to plan a path of the grasping section for handing over theobject to a recipient by the handing-over area part; and a grasp controlsection configured to control grasp operation of the object by thegrasping section in accordance with the planned path.

In the above-described embodiment, the robot apparatus may furtherinclude an imaging section, wherein the recognition section configuredto recognize the graspable part and the handing-over area part of theobject on the basis of an image recognition result of a captured imageof the object by the imaging section.

In the above-described embodiment, the recognition section may beconfigured to recognize a part enabling a user who receives the objectto easily receive the object as the handing-over area part, andrecognize a part allowed to be grasped by the grasping section, butdifficult for the user to receive as the graspable part on the basis ofan image recognition result of a captured image of the object by theimaging section.

In the above-described embodiment, the recognition section may beconfigured to further recognize a part of the object not allowed to beheld by the grasping section as an ungraspable part.

In the above-described embodiment, the grasp planning section may beconfigured to plan a path of the grasping section for handing over bygrasping the object by the graspable part, and handing over the objectby the handing-over area part.

In the above-described embodiment, if the grasp planning section failsto plan a path of the grasping section for handing over the graspablepart of the object in a good posture, the grasp planning section may beconfigured to plan a path of the grasping section for changing to holdthe graspable part of the object in a good posture, and then to handover the object to the recipient by the handing-over area part.

In the above-described embodiment, if the grasp planning section failsto plan a path of the grasping section for handing over the graspablepart of the object in a good posture, the grasp planning section may beconfigured to plan a path of the grasping section for grasping theobject by the handing-over area part, and then changing to hold thegraspable part of the object in a good posture, and then handing overthe object by the handing-over area part.

In the above-described embodiment, if the grasp planning section failsto grasp the graspable part of the object in a good posture, the graspplanning section may be configured to plan a path of the graspingsection for grasping the object by the handing-over area part, andhanding over the object to the recipient by the graspable part.

In the above-described embodiment, when the grasp planning section failsto grasp the graspable part of the object in a good posture even ifchanging to hold the object, the grasp planning section may beconfigured to plan a path of the grasping section for grasping theobject by the handing-over area part, and handing over the object to therecipient by the graspable part.

In the above-described embodiment, the grasp planning section may beconfigured to give a warning to the recipient to whom the object ishanded over by the graspable part.

In the above-described embodiment, if the grasp planning section failsto plan a path of the grasping section for handing over the objecteither by the handing-over area part or by the graspable part, the graspplanning section may be configured to give a warning of being unable tohand over the object to the recipient.

According to an embodiment of the present disclosure, there is provideda method of controlling a robot apparatus, including: recognizing agraspable part and a handing-over area part of an object to be graspedby a grasping section; grasp planning a path of the grasping section forhanding over the object to a recipient by the handing-over area part;and controlling grasp operation of the object by the grasping section inaccordance with the planned path.

According to an embodiment of the present disclosure, there is provideda computer program described in a computer-readable format for causing acomputer to function as an apparatus including: a recognition sectionconfigured to recognize a graspable part and a handing-over area part ofan object to be grasped by a grasping section; a grasp planning sectionconfigured to plan a path of the grasping section for handing over theobject to a recipient by the handing-over area part; and a grasp controlsection configured to control grasp operation of the object by thegrasping section in accordance with the planned path.

The computer program described above is a computer program defined in acomputer-readable format such that predetermined processing is achievedon a computer. To put it in another way, by installing theabove-described computer program on a computer, cooperative operation isperformed on the computer. Accordingly, it is possible to obtain a sameworking effect as that of the robot apparatus described above.

By the technique disclosed in this specification, it is possible toprovide an excellent robot apparatus, a method of controlling the robotapparatus, and a computer program, which is capable of performinghanding-over interaction of an object with a user without imposingexcessive load on the user and without endangering the user.

By the technique disclosed in this specification, it is possible toachieve object handing-over interaction from a robot apparatus to aperson safely and efficiently in consideration of easiness of receivingby a recipient of the object.

Also, by the technique disclosed in this specification, if determinedthat it is difficult to hand over an object, it is possible for therobot apparatus to inform a recipient of that in advance so as to reduceuncomfortable feeling given to the recipient.

Also, by the technique disclosed in this specification, it is possibleto achieve handing-over interaction between a robot apparatus and aperson at a low cost without using a device, such as a wireless tag, areading device, etc.

The other objects, features, and advantages of the technique disclosedin this specification will further become apparent by a detaileddescription based on an embodiment described below and accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an outer view of a robot apparatus towhich a technique disclosed in this specification can be applied;

FIG. 2 is a diagram schematically illustrating a configuration ofdegrees of freedom in joints of the robot apparatus to which a techniquedisclosed in this specification can be applied;

FIG. 3 is a diagram schematically illustrating a configuration of acontrol system of the robot apparatus illustrated in FIG. 1;

FIG. 4 is a diagram illustrating an example in which scissors aredivided into areas;

FIG. 5 is a diagram illustrating an example in which a plastic bottle isdivided into areas;

FIG. 6 is a diagram illustrating an example in which a mug is dividedinto areas;

FIG. 7 is a diagram illustrating an example in which a knife is dividedinto areas;

FIG. 8A is a flowchart illustrating a processing procedure in order forthe robot apparatus to hand over an object;

FIG. 8B is a flowchart illustrating a processing procedure in order forthe robot apparatus to hand over an object;

FIG. 9 is a diagram illustrating a state in which a grasp objectcoordinate system Σobj1 is given to a graspable part of a mug, which isan object to be grasped;

FIG. 10 is a diagram illustrating a state in which a grasping coordinatesystem Σhd is given to a grasping section used by the robot apparatusfor grasping an object;

FIG. 11 is a diagram illustrating an example in which a graspable partof a mug is replaced by a primitive shape (cylinder);

FIG. 12 is a diagram illustrating a state in which the graspingcoordinate system Σhd given to the grasping section of the robotapparatus is overlaid on the grasp object coordinate system Σobj1 givento the graspable part of the mug;

FIG. 13 is a diagram illustrating a state in which a grasp objectcoordinate system Σobj2 is given to a handing-over area part of a mug,which is an object to be grasped;

FIG. 14 is a diagram illustrating a relationship between a handing-overcoordinate system Σrv given to a palm of a user's hand, and grasp objectcoordinate systems Σobj1 and Σobj2 given to a graspable part and ahanding-over area part, respectively, of a knife which is an object tobe handed over;

FIG. 15 is a diagram illustrating a relationship between a handing-overcoordinate system Σrv given to a palm of a user's hand, and grasp objectcoordinate systems Σobj1 and Σobj2 given to a mug which is an object tobe handed over;

FIG. 16 is a diagram illustrating a state in which a grasp objectcoordinate system Σobj1 is given to a puzzle, which is an object thatmakes it necessary to maintain a posture while handing-over operation isperformed;

FIG. 17 is a diagram illustrating a state in which a grasp objectcoordinate system Σobj1 is given to a plate full of dishes, which is anobject that makes it necessary to maintain a posture while handing-overoperation is performed;

FIG. 18 is a flowchart illustrating a processing procedure for handingover an object to a user by an easy way to hand over; and

FIG. 19 is a flowchart illustrating a functional configuration of therobot apparatus that hands over an object to a user.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following, a detailed description will be given of an embodimentof the technique disclosed in this specification with reference to thedrawings.

FIG. 1 illustrates an outer view of a robot apparatus 100 to which thetechnique disclosed in this specification can be applied. The robotapparatus 100 has a link structure in which a plurality of links areconnected by joints, and the individual joints are operated byactuators, respectively. Also, FIG. 2 schematically illustrates aconfiguration of degrees of freedom in joints of the robot apparatus100. The robot apparatus 100 illustrated in FIG. 1 and FIG. 2 isinstalled in a surrounding environment that is dynamically changingevery moment, such as in a home, etc., and performs life support, suchas household chores, nursing care, etc. However, it is also possible forthe robot apparatus 100 to be installed in a factory, etc., and to carryout a fixed task.

The robot apparatus 100 illustrated in FIG. 1 and FIG. 2 is a dual-armtype, and is provided with two driving wheels 101R and 101L that areopposed to each other in a base section as a moving means. The drivingwheels 101R and 101L are driven by driving wheel actuators 102R and 102Lthat individually rotate around a pitch axis, respectively. In thisregard, in FIG. 2, reference numerals 151, 152, 153 are underactuatedjoints that do not actually exist, and correspond to a translationaldegree of freedom in an X direction (forward and backward directions), atranslational degree of freedom in a Y direction (right and leftdirections), and a rotational degree of freedom around a yaw axis,respectively, of the robot apparatus 100 with respect to a floor face,thereby representing movement of the robot apparatus 100 in a virtualworld.

The moving means is connected to an upper body through a waist joint.The moving means is driven by a waist-joint pitch axis actuator 103 thatrotates around a pitch axis. The upper body includes two, right andleft, arm sections and a head section connected through a neck joint.Each of the right and the left arm sections has three degrees of freedomat the shoulder joint, two degrees of freedom at an elbow joint, and twodegrees of freedom at a wrist joint, and thus seven degrees of freedomin total. The three degrees of freedom at a shoulder joint are driven byshoulder-joint pitch axis actuators 104R/L, shoulder-joint roll axisactuators 105R/L, and shoulder-joint yaw axis actuators 106R/L. The twodegrees of freedom at the elbow joint are driven by elbow-joint pitchaxis actuators 107R/L and elbow-joint yaw axis actuators 108R/L. The twodegrees of freedom at the wrist joint are driven by wrist-joint rollaxis actuators 109R/L and wrist-joint pitch axis actuators 110R/L. Also,the two degrees of freedom at the neck joint are driven by a neck-jointpitch axis actuator 111 and a neck-joint roll axis actuator 112. Also,one degree of freedom at a hand joint at an end of right or left armsection is driven by a corresponding one of hand-joint roll axisactuators 113R/L. In the present disclosure, it is possible to achievegrasp operation of an object with a hand by driving the hand-joint rollaxis actuators 113R/L.

In this regard, although the robot apparatus 100 illustrated in FIG. 1and FIG. 2 is provided with a moving means having opposed two wheels,the gist of the technique disclosed in this specification is not limitedto a moving means having opposed two wheels. For example, it is alsopossible to apply the technique disclosed in this specification in thesame manner to a robot apparatus provided with a moving means havinglegs, or a robot apparatus provided without a moving means.

An encoder for measuring a joint angle, a motor for generating torque, acurrent-controlled type motor driver for driving the motor, and areduction gear for obtaining sufficient generated force are attached toan actuator of each axis. Also, each actuator is provided with amicrocomputer that performs drive control of the actuator (both are notillustrated in FIG. 2).

Dynamics computing of the robot apparatus 100 is performed on a hostcomputer (not illustrated in the figure), for example, and controltarget values of torque or joint angles of the individual actuators aregenerated. The control target values are transmitted to the controlmicrocomputer provided to the actuators, and used for controlling theactuators performed by the control microcomputers, respectively. Also,the individual joint actuators are controlled by a force control methodor a position control method.

FIG. 3 schematically illustrates a functional configuration forcontrolling the robot apparatus 100 illustrated in FIG. 1. The robotapparatus 100 includes a control unit 310, which performs overallcontrol of entire operation and the other data processing, aninput/output section 320, and a drive section 330. In the following,descriptions will be given of the individual sections.

The input/output section 320 includes, as an input section, a camera 321and a three-dimensional sensor 322 corresponding to eyes of the robotapparatus 100 and a microphone 323 corresponding to ears of the robotapparatus 100, etc. Also, the input/output section 320 includes, as anoutput section, a speaker 324 corresponding to a mouth of the robotapparatus 100, etc. Here, the camera 321 includes an image sensingdevice, such as a CCD (Charge Coupled Device) or a CMOS (ComplementaryMetal Oxide Semiconductor), etc. Also, the three-dimensional sensor 322is a device capable of measuring a three dimensional position andposture of an object, and includes, for example, a stereo camera, alaser range finder, Kinect (registered trademark), etc. Also, the robotapparatus 100 is allowed to receive an instruction of a task, forexample, by audio input of a user from the microphone 323. However, therobot apparatus 100 may include an input means (not illustrated in thefigure) of an instruction of the other tasks through a wired connection,a wireless connection, or through a recording medium, etc.

Also, the input/output section 320 includes a wireless communicationsection 325. The robot apparatus 100 is allowed to perform datacommunication with a user's instruction terminal (a tablet terminal,etc., not illustrated in FIG. 3), etc., by the wireless communicationsection 325 through a communication path, such as Wi-Fi, etc.

The drive section 330 is a functional module for achieving degrees offreedom at each joint of the robot apparatus 100, and includes aplurality of drive units disposed for each axis, such as a roll axis, apitch axis, and a yaw axis, etc., at the individual joints. Each of thedrive units includes a combination of a motor 331 performing rotationaloperation around a predetermined axis, an encoder 332 detecting arotational position of the motor 331, and a driver 333 adaptivelycontrolling a rotational position and a rotational velocity of the motor331 on the basis of an output of the velocity encoder 332.

The control unit 310 includes a recognition section 311, a drive controlsection 312, and an environment map 313.

The recognition section 311 recognizes a surrounding environment on thebasis of information obtained from the input section, such as the camera321 and the range sensor 322 in the input/output section 320. Forexample, the recognition section 311 builds and updates the environmentmap 313 in advance on the basis of position and posture information ofthe camera 321 that is obtained by own-position estimation processingestimating a position of the camera 321, and object information that isobtained by image recognition processing detecting an object from acaptured image of the camera 321.

The drive control section 312 controls the output section out of theinput/output section 320, and driving of the drive section 330. Forexample, the drive control section 312 performs control of the drivesection 330 in order for the robot apparatus 100 to achieve tasks. Thetasks achieved by the robot apparatus 100, mentioned here, includephysical interaction with a user, such as handing over of an objectinstructed by the user, etc. In order to achieve the interaction, therobot apparatus 100 drives the driving wheel actuators 102R and 102L,and the individual joint actuators of an arm section.

The robot apparatus 100 is applied to, for example, life support, andbrings a desired object for a physically handicapped person, etc., inorder to hand over the object to the person. However, even if a robotapparatus grasps an object in an easy way to receive (or an easy postureof an object to receive), and tries to hand over the object, a recipientuser sometimes finds it difficult to receive the object. Thus, in thisembodiment, object handing-over interaction from the robot apparatus 100to a person is made safe and efficient in consideration of easiness ofreceiving by a recipient of the object. Also, if determined that it isdifficult to hand over the object, the robot apparatus 100 informs therecipient of the difficulty in advance so as to reduce uncomfortablefeeling given to the recipient.

In this embodiment, grasping areas of individual objects to be graspedare divided into areas of a graspable part, a handing-over area part,and an ungraspable part in consideration of easiness to receive at thetime of handover. The handing-over area part is a part of the objectallowed for a user who receives the object to receive easily. Thegraspable part is an easy part for the robot apparatus 100 to grasp, buta difficult part for the user to receive. The ungraspable part is a partthat is difficult bot for the robot apparatus 100 and the user to grasp.Such area information for each object category may be put into adatabase.

FIG. 4 illustrates an example in which scissors, an example of an objectto be grasped, are divided into areas of a graspable part 401, ahanding-over area part 402, and an ungraspable part 403. As illustratedin the figure, a handle becomes the handing-over area part 402, a backbecomes the graspable part 401, and a strap attached to the handle isflexible to be held (or it is difficult to be held in order to hold aposition of the handing-over area part 402), and thus becomes theungraspable part 403. In this regard, the other examples of the areas,which are to be handled as ungraspable parts, include an adhesive areadisposed on an object surface, and an area susceptible to damage, suchas a lens which is susceptible to be hurt by just being touched, forexample.

Also, FIG. 5 illustrates an example in which a plastic bottle is dividedinto a graspable part 501 and a handing-over area part 502. Asillustrated in FIG. 5, a shoulder part and a bottom part of the plasticbottle becomes the graspable part 501, and a body part in the middle ofthe bottle becomes a handing-over area part 502. Also, FIG. 6illustrates an example in which a mug is divided into a graspable part601 and a handing-over area part 602. As illustrated in FIG. 6, acylindrical main body part of the cup becomes a graspable part 601, anda handle part becomes a handing-over area part 602. Also, FIG. 7illustrates an example in which a knife is divided into a graspable part701 and a handing-over area part 702. As illustrated in FIG. 7, a bladepart becomes a graspable part 701, and a handle part becomes ahanding-over area part 702.

If the robot apparatus 100, which hands over an object, holds the objectby a graspable part, and a user, who receives the object, receives theobject by a handing-over area part, it is easy for the user to receive,and thus it is efficient and safe work. However, if it is difficult forthe robot apparatus 100 to grasp the graspable part of the objectbecause of a shape, a position, a posture, etc., of the object, therobot apparatus 100 verifies whether it is possible to grasp thehanding-over area part. And if possible, the robot apparatus 100 graspsthe object by the handing-over area part in place of the graspable part.

If the robot apparatus 100 has dual arms, it is possible to change bothhands to hold the object, whereas if the robot apparatus 100 has asingle hand, the robot apparatus 100 may temporarily places the object,and then may change holding the object. If the robot apparatus 100 hasgrasped an object to hand over by a handing-over area part, the robotapparatus 100 verifies whether it is possible to change holding from ahanding-over area part to a graspable part. And if possible, the robotapparatus 100 changes to hold a graspable part in a good posture, andthen hands over the object to a recipient. Also, if it is difficult tochange holding the object, the robot apparatus 100 sends a message thatit is difficult to hand over to the user, and tries to hand over theobject by the handing-over area part in a good posture, or to hand overthe object by the graspable part to the user. Further, if it is notpossible for the robot apparatus 100 to grasp the object by thehanding-over area part, the robot apparatus 100 sends a message that itis not possible to grasp, or to hand over to the user.

In this regard, the “user” mentioned here corresponds to a “receivingside”, and includes a receiving robot, etc., in addition to the case ofa receiving person.

Also, it is possible to send a message to the user, for example, byoutputting a voice message from the speaker 324, but it is also possibleto inform the user by another way instead.

FIG. 19 illustrates a functional configuration of the robot apparatus100 that hands over an object to a user. The robot apparatus 100illustrated in FIG. 19 includes a control section 1, an image processingsection 2, an imaging section 3, an object recognition section 4, agrasp planning section 5, a grasp control section 6, and a graspingsection 7.

The control section 1, the image processing section 2, the objectrecognition section 4, the grasp planning section 5, and the graspcontrol section 6 correspond to the control unit 310 in FIG. 3. Thecontrol section 1 totally controls the overall operation of the robotapparatus 100. The image processing section 2 and the object recognitionsection 4 correspond to the recognition section 311, and the graspplanning section 4 and the grasp control section 5 correspond to thedrive control section 312. Also, the imaging section 3 corresponds tothe camera 321, and the grasping section 7 mainly includes dual arms ofthe robot apparatus 100.

The imaging section 3 captures an image of an object to be handed overto the user, and a palm of the user who is a handing-over recipient,etc. The image processing section 2 performs image processing on thecaptured image by the imaging section 3.

The object recognition section 4 recognizes an object to be handed overto the user from an image processing result of the captured image, anddivides the object into a graspable part, a handing-over area part, andan ungraspable part, etc. Also, the object recognition section 4 givesgrasping coordinate systems Σobj1 and Σobj2 (described later) to thegraspable part and the handing-over area part, respectively. Also, theobject recognition section 4 recognizes a palm of the user who is arecipient of the object from the image processing result of the capturedimage, and gives a handing-over coordinate system Σhd (described later).When the object recognition section 4 performs processing to divide anarea and to give the coordinate system as described above, the objectrecognition section 4 may refer to an object database 8. The database 8stores area information for each object category, original points andposture of the grasping coordinate systems given to the individualareas, ranges of areas, hand-over vectors (described later), etc.

The grasp planning section 5 makes a path plan of a position and postureof the grasping section 7 for handing over the object to the user on thebasis of a recognition result, such as the object and a palm of user'shand, etc., by the object recognition section 4.

The grasp planning section 5 plans a path of a position and posture ofthe grasping section 7 so as to hold the graspable part of the objectbasically by the grasping section 7, and hand over the object to theuser by the handing-over area part of the object. However, if it isdifficult to grasp the graspable part of the object by the graspingsection 7 because of a shape, a position, a posture, etc., of theobject, the grasp planning section 5 plans the path flexibly. Forexample, the grasp planning section 5 plans to temporarily hold thehanding-over area part of the object, and after that, to hold thegraspable part of the object instead, and to hand over the handing-overarea part of the object to the user. Alternatively, the grasp planningsection 5 plans to hand over the graspable part of the object to theuser, etc. Also, if it is not possible for the grasp planning section 5to plan a path of the grasping section 7, which is easy for the user toreceive the object, or if it is difficult for the grasp planning section5 to plan a path of the grasping section 7 for handing over the objectin the first place, the grasp planning section 5 sends a message statingthat to the user.

In this regard, any method may be employed for a method of determining aposture of the grasping section 7 at the time of grasping an object. Forexample, it is possible to apply a method disclosed in Japanese PatentApplication No. 2011-262202 specification, which has been alreadyassigned to the present applicant.

And the grasp control section 6 controls the operation of the graspingsection 7 in accordance with the planned path to achieve handing overthe object to the user. In this regard, any method may be employed for amethod of controlling the operation of the grasping section 7 inaccordance with the determined grasp posture. For example, it ispossible to apply a method disclosed in Japanese Patent Application No.2011-262202 specification, which has been already assigned to thepresent applicant.

FIG. 8A and FIG. 8B illustrate a processing procedure for the robotapparatus 100 to hand over an object in a format of a flowchart.

First, a flag indicating that an object is grasped by a graspable partis turned off (step S801).

Next, an image of an object to be handed over is recognized in thecaptured image of the camera 321 (step S802), and the object isidentified (step S803).

Next, an area of the object included in the captured image is dividedinto a graspable part, a handing-over area part, and an ungraspable part(step S804), and the ungraspable part is excluded from the target ofgrasp (step S805).

Next, whether it is possible to grasp this object by the graspable partor not is verified from the shape of the object identified by step S803,and the current position and posture of the object (step S806).

Here, if it is possible to grasp the object to be handed over by thegraspable part (Yes in step S806), the flag indicating that the objectis grasped by the graspable part is turned on (step S807), and it isfurther determined whether it is possible for the robot apparatus 100 tograsp the graspable part of the object in a good posture or not (stepS808).

And if it is possible to grasp the object by the graspable part in agood posture (Yes in step S808), the robot apparatus 100 grasps theobject by the graspable part (step S809), and hands over the object inan easy way for the user to receive (step S810).

On the other hand, if it is not possible for the robot apparatus 100 tograsp the object to be handed over by the graspable part (No in stepS806), and it is not possible to grasp the object by the graspable partin a good posture (No in step S808), a determination is made of whetherit is possible to grasp the object by the handing-over area part inplace of the graspable part in a good posture or not (step S811).

If it is possible to grasp the handing-over area part of the object in agood posture (Yes in step S811), the robot apparatus 100 grasps theobject by the handing-over area part for the time being (step S812).After that, a determination is made of whether it is possible for therobot apparatus 100 to change to hold the object by the graspable partin a good posture (step S813).

If it is possible for the robot apparatus 100 to change to hold theobject by the graspable part in a good posture (Yes in step S813), therobot apparatus 100 changes to hold the object by the graspable part ina good posture (step S814), and then to hand over the object to the userin an easy way to receive (step S810).

Also, if it is not possible for the robot apparatus 100 to change tohold the object by the graspable part in a good posture (No in stepS813), the robot apparatus 100 sends a message to the user that it isdifficult to hand over the object, and then carries out the handing-overof the object (step S815).

Also, if it is not possible for the robot apparatus 100 to grasp thehanding-over area part of the object in a good posture (No in stepS811), a determination is made of whether the flag is set on or not(step S816).

Here, a state of the flag being on (Yes in step S816) indicates that itis possible to currently grasp the graspable part of the object in anincorrect posture, and thus the robot apparatus 100 grasps the graspablepart of the object in the incorrect posture (step S817). After that, therobot apparatus 100 determines whether it is possible to change to holdthe object by the graspable part in a good posture (step S813). If it ispossible to change to hold the object by the graspable part (Yes in stepS813), the robot apparatus 100 changes to hold the object by thegraspable part in a good posture (step S814), and hands over the objectto the user in an easy way to hand over (step S810). Also, if it is notpossible for the robot apparatus 100 to change to hold the object by thegraspable part in a good posture (No in step S813), the robot apparatus100 sends a message to the user that it is difficult to hand over theobject, and carries out the handing-over of the object (step S815).

Also, a state of the flag not being on (No in step S816), indicates thatit is not possible for the robot apparatus 100 to grasp the graspablepart of the object (even in an incorrect posture), and that it is notpossible to grasp the handing-over area part in a good posture. In thiscase, a determination is made of whether it is possible for the robotapparatus 100 to grasp the handing-over area part of the object (if notin a good posture) (step S818).

If it is possible for the robot apparatus 100 to grasp the handing-overarea part of the object (if not in a good posture) (Yes in step S818),the robot apparatus 100 grasps the handing-over area part of the objectfor the time being (step S819). After that, a determination is made ofwhether it is possible for the robot apparatus 100 to change to hold theobject by the graspable part in a good posture (step S813). And if it ispossible for the robot apparatus 100 to change to hold the object by thegraspable part in a good posture (Yes in step S813), the robot apparatus100 changes to hold the object by the graspable part in a good posture(step S814), and then hands over the object to the user in an easy wayto hand over (step S810). Also, if it is not possible for the robotapparatus 100 to change to hold the object by the graspable part in agood posture (No in step S813), the robot apparatus 100 sends a messageto the user that it is difficult to hand over the object, and thencarries out the handing-over of the object (step S815).

Also, if it is not possible for the robot apparatus 100 to grasp theobject either by the graspable part or by the handing-over area part (Noin step S818), the robot apparatus 100 sends a message that it is notpossible to grasp the object (step S820), and then terminates thisprocessing.

In this regard, if the robot apparatus 100 includes two arms as in thisembodiment (or three arms or more), it is possible to change to hold theobject using both hands in the case of dual arms. Also, in the case of arobot apparatus with a single arm, it is possible to temporarily placethe object by one hand, and then to change holding.

FIG. 9 illustrates a state in which a grasp object coordinate systemΣobj1 is given to a graspable part of a mug, which is an object to begrasped. The grasp object coordinate system Σobj1 is positioned at acenter point of the object, and represents a good posture when theobject is handed over to the user.

Also, FIG. 10 illustrates a state in which a grasping coordinate systemΣhd is given to a grasping section used by the robot apparatus 100 forgrasping an object. The grasping section illustrated in FIG. 10 is justlike a gripper, and a size of a graspable object is determined inaccordance with a width of the gripper, etc. In FIG. 10, the graspingcoordinate system Σhd is given to a grasp center point of the graspingsection, and a direction in which the grasping section tries to grab theobject, that is to say a hand-over direction vector X, is expressed.

When the robot apparatus 100 recognizes an image of a mug, which is anobject to be grasped, from an image captured by the camera 321, thegrasp object coordinate system Σobj1 is given to a center point of thegraspable part of the mug as illustrated in FIG. 9. Also, in order tosimplify the processing for determining whether graspable or not, thegraspable part is replaced by a primitive shape by the image processing.The primitive shape mentioned here is a simple geometrical shape, suchas a cylinder, a polygonal prism, a circular cone, a polygonal pyramid,etc., for example. FIG. 11 illustrates an example in which a graspablepart of a mug is replaced by a cylinder.

In step S806 in the flowchart illustrated in FIG. 8A, a comparison ismade between a primitive shape and a graspable size of the graspingsection. If the primitive shape is smaller than the graspable size ofthe grasping section, a determination is made that the graspable part ofthe object is graspable.

Also, FIG. 12 illustrates a state in which the grasping coordinatesystem Σhd given to the grasping section of the of the robot apparatus100 is overlaid on the grasp object coordinate system Σobj1 given to thegraspable part of the mug. In step S808 in the flowchart illustrated inFIG. 8A, if it is possible to make the grasp object coordinate systemΣobj1, or a hand-over vector described later, and each element vector(x, y, z) of the grasping coordinate system Σhd not greater than aposture value given to the mug, a determination is made that thegraspable part is graspable in a good posture.

Also, FIG. 13 illustrates a state in which a grasp object coordinatesystem Σobj2 is given to a handing-over area part of a mug, which is anobject to be grasped. In step S811 in the flowchart illustrated in FIG.8A, if it is possible to make the grasp object coordinate system Σobj2,or the hand-over vector described later, and each element vector (x, y,z) of the grasping coordinate system Σhd not greater than a posturevalue given mug, a determination is made that the handing-over area partis graspable in a good posture.

Also, if the robot apparatus 100 recognizes an image of a palm of a userwho is a recipient of the object from the image captured by the camera321, the robot apparatus 100 gives the handing-over coordinate systemΣrv to a center point thereof. However, if a sensor capable of detectinga posture, such as a gyroscope, etc., is attached to the palm of theuser, it is possible to give the handing-over coordinate system Σrv onthe basis of a sensor value thereof.

At the time of carrying out the handing over of the object, the robotapparatus 100 hands over the object in a manner that is close to avector formed by a vector given to the grasp object coordinate systemsΣobj1 or Σobj2 (Σobj1 in the case of grasping the object by thegraspable part, whereas Σobj2 in the case of grasping the object by thehanding-over area part) and the handing-over coordinate system Σrv. FIG.14 illustrates a relationship between a handing-over coordinate systemΣrv given to a palm of a user, and grasp object coordinate systems Σobj1and Σobj2 respectively given to a graspable part and a handing-over areapart of a knife, which is an object to be handed over.

Also, FIG. 15 illustrates a relationship between a handing-overcoordinate system Σrv given to given to a palm of a user, and graspobject coordinate systems Σobj1 and Σobj2 given to a mug, which is anobject to be handed over. If an object to be handed over is a mug, andthe mug contains drink, it is necessary to add a restrictive conditionthat the mug should be kept horizontal during handing-over operation,that is to say, the grasp object coordinate system should be keptconstant, which is different from the case of a pen. As illustrated inFIG. 15, for such an object, a “hand-over vector” indicating a directionof handing over the object and a “vertical direction vector” indicatinga vertical direction of the object are given to the grasp objectcoordinate system Σobj1. And handing over is carried out in a posturedetermined by the hand-over vector and the vertical direction vectorregardless of the handing-over coordinate system Σrv. The hand-overvector may be one vector, a plurality of vectors, or face definition maybe made.

Also, another examples of the case where it is necessary to keep postureof an object while the handing-over operation is performed may include apuzzle (refer to FIG. 16), and a plate on which food is served (refer toFIG. 17). FIGS. 16 and 17 illustrate a state in which a grasp-objectcoordinate system Σobj1 is given. In the case of the puzzle, it isnecessary that the vertical direction is correct, horizontal conditionis kept, and the hand-over direction is a certain direction, and thus aplurality of hand-over vectors are given. On the other hand, in the caseof the plate on which food is served, as long as the vertical directionis correct, and horizontal condition is kept, it is sufficient, and thushand-over vectors that define faces are given, and handing over iscarried out in the vector posture irrelevantly to the handing-overcoordinate system Σrv.

As already described, area information on a graspable part for eachobject category, a handing-over area part, and an ungraspable part isput into a database. In this database, an origin position and posture ofthe grasping coordinate system given to each area, an area range, ahand-over vector are stored for each object category in addition to thearea information. As in the case of a glass with drink, in the case ofan object that is necessary to determine handing-over postureirrelevantly to the handing-over coordinate system Σrv of the user,hand-over vectors are necessary.

FIG. 18 illustrates, in a flowchart form, a processing procedure forhanding over an object to a user by an easy way to hand over using agrasping coordinate system and a hand-over vector for each area set onthe object in step S810 in the flowchart illustrated in FIG. 8A.

First, a determination is made whether the robot apparatus 100 isgrasping an object along the hand-over vector (step S1801).

Here, if the robot apparatus 100 is grasping the object along thehand-over vector (Yes in step S1801), the robot apparatus 100 performshanding-over operation while maintaining the current grasping postureirrelevantly to the handing-over coordinate system Σrv of the user, whois a recipient (step S1802).

On the other hand, if the robot apparatus 100 is not grasping the objectalong the hand-over vector (No in step S1801), the robot apparatus 100confirms the handing-over coordinate system Σrv of the user, who is therecipient (step S1803). The robot apparatus 100, for example, recognizesan image of a palm of the user, who is the recipient, from the imagecaptured by the camera 321, and confirms the handing-over coordinatesystem Σrv at the center point thereof. Alternatively, if the robotapparatus 100 recognizes a palm of the user, who is the recipient, fromthe image captured by the camera 321, the robot apparatus 100 confirmsthe handing-over coordinate system Σrv at a center point thereof.

And robot apparatus 100 hands over the object such that an X-axisdirection of the grasping coordinate system Σhd of the grasping section(that is to say, the handing-over direction vector) matches a vectorformed by the grasp object coordinate system Σobj2 of the object(handing-over area part) and the handing-over coordinate system Σrv ofthe user (step S1804).

In this manner, by the present embodiment, it is possible to achieveobject handing-over interaction from the robot apparatus 100 to a personsafely and efficiently in consideration of easiness of receiving by arecipient of an object. Also, if determined that it is difficult to handover an object, it is possible for the robot apparatus 100 to reduceuncomfortable feeling given to a recipient by informing the recipient ofthat in advance. In this regard, it is possible to configure thetechnique disclosed in this specification as follows.

-   -   (1) A robot apparatus including: a grasping section configured        to grasp an object; a recognition section configured to        recognize a graspable part and a handing-over area part of the        object; a grasp planning section configured to plan a path of        the grasping section for handing over the object to a recipient        by the handing-over area part; and a grasp control section        configured to control grasp operation of the object by the        grasping section in accordance with the planned path.    -   (2) The robot apparatus according to (1), further including an        imaging section, wherein the recognition section is configured        to recognize the graspable part and the handing-over area part        of the object on the basis of an image recognition result of a        captured image of the object by the imaging section.    -   (3) The robot apparatus according to (1), wherein the        recognition section is configured to recognize a part enabling a        user who receives the object to easily receive the object as the        handing-over area part, and recognize a part allowed to be        grasped by the grasping section, but difficult for the user to        receive as the graspable part on the basis of an image        recognition result of a captured image of the object by the        imaging section.    -   (4) The robot apparatus according to (3), wherein the        recognition section is configured to further recognize a part of        the object not allowed to be held by the grasping section as an        ungraspable part.    -   (5) The robot apparatus according to (1), wherein the grasp        planning section is configured to plan a path of the grasping        section for handing over the object to the recipient by grasping        the object by the graspable part, and handing over the object by        the handing-over area part.    -   (6) The robot apparatus according to (5), wherein if the grasp        planning section fails to plan a path of the grasping section        for handing over the graspable part of the object in a good        posture, the grasp planning section is configured to plan a path        of the grasping section for changing to hold the graspable part        of the object in a good posture, and then to hand over the        object to the recipient by the handing-over area part.    -   (7) The robot apparatus according to (5), wherein if the grasp        planning section fails to plan a path of the grasping section        for handing over the graspable part of the object in a good        posture, the grasp planning section is configured to plan a path        of the grasping section for grasping the object by the        handing-over area part, and then changing to hold the graspable        part of the object in a good posture, and then handing over the        object by the handing-over area part.    -   (8) The robot apparatus according to (5), wherein if the grasp        planning section fails to grasp the graspable part of the object        in a good posture, the grasp planning section is configured to        plan a path of the grasping section for grasping the object by        the handing-over area part, and handing over the object to the        recipient by the graspable part.    -   (9) The robot apparatus according to (6) or (7), wherein when        the grasp planning section fails to grasp the graspable part of        the object in a good posture even if changing to hold the        object, the grasp planning section is configured to plan a path        of the grasping section for grasping the object by the        handing-over area part, and handing over the object to the        recipient by the graspable part.    -   (10) The robot apparatus according to (8) or (9), wherein the        grasp planning section is configured to give a warning to the        recipient to whom the object is handed over by the graspable        part.    -   (11) The robot apparatus according to any one of (5) to (9),        wherein if the grasp planning section fails to plan a path of        the grasping section for handing over the object either by the        handing-over area part or by the graspable part, the grasp        planning section is configured to give a warning of being unable        to hand over the object to the recipient.    -   (12) A method of controlling a robot apparatus, including:        recognizing a graspable part and a handing-over area part of an        object to be grasped by a grasping section; grasp planning a        path of the grasping section for handing over the object to a        recipient by the handing-over area part; and controlling grasp        operation of the object by the grasping section in accordance        with the planned path.    -   (13) A computer program described in a computer-readable format        for causing a computer to function as an apparatus including: a        recognition section configured to recognize a graspable part and        a handing-over area part of an object to be grasped by a        grasping section; a grasp planning section configured to plan a        path of the grasping section for handing over the object to a        recipient by the handing-over area part; and a grasp control        section configured to control grasp operation of the object by        the grasping section in accordance with the planned path.

In the above, the detailed description has been given of the techniquedisclosed in this specification with reference to the specificembodiment. However, it is apparent that modifications and substitutionsof the embodiment may be made by those skilled in the art withoutdeparting from the spirit and scope of the technique disclosed in thepresent specification.

In short, a description has been given of the technique disclosed inthis specification by exemplification. However, the description of thisspecification should not be construed in a limited manner. In order todetermine the gist of the technique disclosed in this specification, theappended claims should be considered.

What is claimed is:
 1. An apparatus, comprising: circuitry configuredto: recognize a first part of an object and a second part of the objectbased on a characteristics of the object, wherein the second part isdifferent from the first part, the first part corresponds to a part ofthe object by which the object is holdable, and the second part of theobject corresponds to a part of the object by which the object is handedover to a user; control a holding section to hold the first part of theobject; determine a first path of the holding section based on thesecond part of the object; determine that the holding section is unableto hold the first part of the object in the first path; determine asecond path to hold the second part of the object, wherein the secondpath is determined based on the determination that the holding sectionis unable to hold the first part of the object; and control a holdoperation of the object by the holding section based on the second path.2. The apparatus according to claim 1, wherein the circuitry is furtherconfigured to: detect a receiving position of the user, wherein thereceiving position corresponds to a position where the object is handedover; determine the first path of the holding section based on thesecond part of the object and a relationship between a first coordinatesystem of the holding section and a second coordinate system of thereceiving position; control the holding section to hand over the objectto the user by the second part of the object, wherein the holdingsection is controlled based on the determined first path of the holdingsection; and control the hold operation of the object by the holdingsection based on the first path.
 3. The apparatus according to claim 1,wherein the circuitry is further configured to: detect a receivingposition of the user based on an image of the user; and determine thefirst path of the holding section based on a relationship between afirst coordinate system of the holding section and a second coordinatesystem of the receiving position.
 4. The apparatus according to claim 1,wherein the circuitry is further configured to recognize the first partof the object and the second part of the object based on an image of theobject.
 5. The apparatus according to claim 4, wherein the circuitry isfurther configured to: recognize the second part of the object based onthe image of the object, wherein the second part enables the user toreceive the object; and recognize the first part of the object based onthe image of the object, wherein the first part is allowed to be held bythe holding section.
 6. The apparatus according to claim 5, wherein thecircuitry is further configured to recognize a third part of the object,the third part is restricted to be held by the holding section, and thethird part of the object corresponds to an unholdable part of theobject.
 7. The apparatus according to claim 1, wherein the circuitry isfurther configured to control, based on the first path, the holdingsection to: hold the object by the first part of the object; andhandover the object by the second part of the object.
 8. The apparatusaccording to claim 7, wherein the circuitry is further configured to:determine that the holding section is unable to hold the first part ofthe object in a specific posture; and determine a third path to hold thefirst part of the object based on the determination that the holdingsection is unable to hold the first part of the object in the specificposture.
 9. The apparatus according to claim 8, wherein the circuitry isfurther configured to determine the third path to: hold the object bythe second part of the object; and hand over the object to the user bythe first part of the object.
 10. The apparatus according to claim 7,wherein the circuitry is further configured to: determine the secondpath to hold the object by the second part of the object; change thesecond path to hold the first part of the object in a specific posture;and hand over the object by the second part of the object based on thechanged second path.
 11. The apparatus according to claim 7, wherein thecircuitry is further configured to determine the second path to: holdthe object by the second part of the object; and hand over the object tothe user by the first part of the object.
 12. The apparatus according toclaim 11, wherein the circuitry is further configured to generate awarning to the user.
 13. The apparatus according to claim 7, wherein thecircuitry is further configured to generate, based on a determinationthat the holding section is unable to hand over the object by one of thefirst part or the second part, a warning that the apparatus is unable tohand over the object to the user.
 14. A method of controlling anapparatus, the method comprising: recognizing a first part of an objectand a second part of the object based on a characteristics of theobject, wherein the second part is different from the first part, thefirst part corresponds to a part of the object by which the object isholdable, and the second part of the object corresponds to a part of theobject by which the object is handed over to a user; controlling aholding section to hold the first part of the object; determining afirst path of the holding section based on the second part of theobject; determining that the holding section is unable to hold the firstpart of the object in the first path; determining a second path to holdthe second part of the object, wherein the second path is determinedbased on the determination that the holding section is unable to holdthe first part of the object; and controlling a hold operation of theobject by the holding section based on the second path.
 15. Anon-transitory computer-readable medium having stored thereoncomputer-readable instructions, which when executed by a computer, causethe computer to execute operations, the operations comprising: in anapparatus: recognizing a first part of an object and a second part ofthe object based on a characteristics of the object, wherein the secondpart is different from the first part, the first part corresponds to apart of the object by which the object is holdable, and the second partof the object corresponds to a part of the object by which the object ishanded over to a user; controlling a holding section to hold the firstpart of the object; determining a first path of the holding sectionbased on the second part of the object; determining that the holdingsection is unable to hold the first part of the object in the firstpath; determining a second path to hold the second part of the object,wherein the second path is determined based on the determination thatthe holding section is unable to hold the first part of the object; andcontrolling a hold operation of the object by the holding section basedon the second path.